The present invention has particular application in minimising so-called “ringing” artifacts that can arise in digital images when displayed.
Most image/video coding standards that are used for compressing digital images, such as JPEG and MPEG-1/2/4, use block-based processing for the compression. Visual artifacts, such as blocking noise and ringing noise, occur in decompressed images due to the block-based coding, coarse quantization and coefficient truncation that are used. Ringing noise manifests itself as halos or rings around edges in the displayed image. Contrast adjustment or enhancement of images, particularly digital images, is used in many fields, including enhancing the contrast of a digital image for display by a television receiver or other display device, for printing by a printer, in digital cameras, etc., etc. Contrast enhancement is used to improve the contrast in medical and other images. A number of techniques are known for attempting to remove or minimise the ringing noise in particular.
For example, in the paper “Coding Artifact Reduction Using Edge Map Guided Adaptive and Fuzzy Filter” by Hao Song Kong et al of the Mitsubishi Electric Research Laboratory (Cambridge, Mass., USA) published in 2004, and the corresponding US patent application US-A-2005/0100241, there is disclosed a method for reducing ringing artifacts in images. This is shown schematically in FIG. 1. In the pixel-classification block, a pixel is classified as a smooth pixel, a texture pixel or an edge pixel using thresholds on the local variance. The local variance is measured using blocks of 3×3 neighbouring pixels. If the pixel is classified as an edge pixel, its neighboring pixels are filtered using a fuzzy identity filter.
A disadvantage of this prior art is the use of thresholds on the local variance. The local variance and, therefore, thresholds depend on the contrast level, which includes the lightning conditions of the source environment when the image is taken. This results in misclassification as it is not possible to adjust the thresholds optimally. Another disadvantage of this prior art is that the fuzzy identity filter uses order-statistical information, which is a complex operation and is therefore expensive to implement and in any event may create artifacts such as local flickering since order-statistical information is not linear and may exhibit jumps.
Another method is disclosed in “Content Adaptive Image De-blocking” by Meng Zhao et. al., published in 2004 by IEEE, which is shown schematically in FIG. 2. This method art also uses a classification-based approach. Each (current) pixel is classified in a pixel classification block using adaptive dynamic range coding (ADRC). A pixel in the neighbourhood of the current pixel is transformed into “1” if its value is greater than the neighbourhood mean, and transformed into “0” if is value is less than the neighbourhood mean. After this operation, the transformed 1s and 0s in a pixel's neighbourhood are appended to obtain a binary number, which is used as an index to a filter look-up table. The filter retrieved from the look-up table is used to filter the pixels in that neighbourhood and obtain the new output pixel corresponding to the current pixel. The filter for each class is computed using a training procedure. Original, artifact-free images are compressed using a block-based compression algorithm, such as JPEG 2000. Then the compressed image is decoded to obtain the degraded image in which artifacts have been introduced. Each pixel in the decoded image is classified using ADRC and the filter coefficients are computed using a linear least squares estimation technique.
One disadvantage of this prior art is again related to the classification block. Misclassifications lead to blurring of artifact-free pixels, which is undesirable. To reduce the misclassification rate, training over a large database of images can be done, but this will decrease the quality of the obtained filters. Also, for good performance, the number of classes must be large, which introduces computational complexity and requires more memory resources.
According to a first aspect of the present invention, there is provided a method of adjusting the contrast of an input image formed of pixels in which each pixel has an input brightness level to produce an output image in which at least some of the pixels have an output brightness level that is different from their input brightness level, the method comprising:
obtaining a blurred image corresponding to the input image, the brightness level of at least some of the pixels in the input image being varied to provide the blurred image;
obtaining a ring likelihood for pixels in the input image, the ring likelihood providing a measure of the likelihood that said pixels are non-edge pixels in the neighbourhood of an edge pixel;
producing the output image as a sum of the brightness levels in the input image and the blurred image in dependence on the ring likelihoods.
Thus, a blurred image is obtained which is then added to the input image in manner that depends on the ring likelihoods at each or at least some of the pixels in the input image, thereby reducing the ringing artifacts around edges whilst preserving edges and without affecting other parts of the image (such as smooth regions and texture regions).
The blurred image may be obtained by filtering the brightness levels of the input image with at least one recursive spatially-adaptive edge-preserving filter. In a preferred embodiment, the filter preserves the edges and blurs both sides of the edges by not mixing pixels on one side of the edge with pixels on the other side of the edge.
The filter preferably has recursion coefficients that are adaptive to edge information in the input image. In a preferred embodiment, where there is an edge, the corresponding recursion coefficient is reduced accordingly depending on the relative strength of the edge. As an alternative to using recursion coefficients, another function can be used that decreases with decreasing edge strength, such as an exponential function.
In a preferred embodiment, the ring likelihood for a pixel is the product of (i) the likelihood that said pixel is a non-edge pixel and (ii) the likelihood of the strongest edge in the neighbourhood of said pixel, wherein said edge likelihoods are obtained from said recursion coefficients. Whilst other functions for obtaining the edge likelihoods may be used, such as the output of a Laplacian filter, the recursion coefficients already contain edge information and have already been calculated in this preferred embodiment, and thus this provides an accurate and yet computationally efficient way of obtaining the ring likelihoods.
In a most preferred embodiment, the blurred image is obtained by filtering the brightness levels of the input image with a first recursive spatially-adaptive edge-preserving filter that operates in a first direction across the pixels of the input image, filtering the brightness levels of the input image with a second recursive spatially-adaptive edge-preserving filter that operates in a second direction across the pixels of the input image that is opposite the first direction, and obtaining the brightness levels of the blurred image by taking an average of the output of the first and second recursive spatially-adaptive edge-preserving filters. As above, in a preferred embodiment, the filters, which may be “forward” and “backward” filters that operate respectively in a forward and a backward direction across the image, preserve the edges and blur both sides of the edges by not mixing pixels on one side of the edge with pixels on the other side of the edge. As is known per se, a filter of this type can give rise to an unwanted phase delay. By using two filters which are run in opposite directions across the pixels, and by averaging the outputs of the two filters, the phase delays of the two filters can effectively be made to cancel each other out.
At least one of the filters preferably has recursion coefficients that are adaptive to edge information in the input image. More preferably, both of the filters have recursion coefficients that are adaptive to edge information in the input image. Again, in a preferred embodiment, where there is an edge, the corresponding recursion coefficient is reduced accordingly depending on the relative strength of the edge.
In a preferred embodiment, the ring likelihood for a pixel is the product of (i) the likelihood that said pixel is a non-edge pixel and (ii) the likelihood of the strongest edge in the neighbourhood of said pixel, wherein said edge likelihoods are obtained from said recursion coefficients. Again, the recursion coefficients already contain edge information and have already been calculated in this preferred embodiment, and thus this provides an accurate and yet efficient way of obtaining the ring likelihoods.
In general, it is preferred that said recursion coefficients decrease with decreasing edge strength.
In embodiments, the ring likelihood for a pixel is the product of (i) the likelihood that said pixel is a non-edge pixel and (ii) the likelihood of the strongest edge in the neighbourhood of said pixel.
Said neighbourhood may be a block of j×k pixels centred on the respective pixel, where j and k are positive integers. In one particular example, the neighbourhood is a block of 8×8 pixels, which corresponds to the blocks of 8×8 pixels used in many block-based compression techniques. Neighborhoods of pixels of other sizes may of course be used, particularly if the original compression that produced the input image used blocks of a different size.
In a preferred embodiment, the sum of the brightness levels in the input image and the blurred image is a weighted sum, the weight of the brightness levels of the blurred image that are summed with the brightness levels of the input image at any particular pixel depending on the ring likelihood for that pixel. In the preferred embodiment, if a pixel is less likely to be a ringing artifact pixel, then less weight is given to the blurred image such that the input image pixels are substantially preserved for artifact-free pixels. On the other hand, if a pixel is more likely to be a ringing artifact pixel, then more weight is given to the blurred image such that the ringing artifact is reduced in this region of the output image.
According to a second aspect of the present invention, there is provided apparatus for adjusting the contrast of an input image formed of pixels in which each pixel has an input brightness level to produce an output image in which at least some of the pixels have an output brightness level that is different from their input brightness level, the apparatus comprising:
an image blurrer for obtaining a blurred image corresponding to the input image, the image blurrer being operable such that the brightness level of at least some of the pixels in the input image are varied to provide the blurred image;
a ring likelihood calculator for obtaining a ring likelihood for pixels in the input image, the ring likelihood providing a measure of the likelihood that said pixels are non-edge pixels in the neighbourhood of an edge pixel; and,
a summer for obtaining a sum of the brightness levels in the input image and the blurred image to produce the output image in dependence on the ring likelihoods.
The preferred apparatus and/or method may be incorporated into any apparatus and/or method that is used to enhance the resolution of a digital image, including for example an image processor used in a television set or the like, printers, digital cameras, television broadcast capture cards, digital image processing software which may be used in many applications, etc., etc. The methods described herein may be carried out by appropriate software running on appropriate computer equipment. The software may be embedded in an integrated circuit, the integrated circuit being adapted for performing, or for use in the performance of, the relevant processes. Many of the processing steps may be carried out using software, dedicated hardware (such as ASICs), or a combination.